Heads up! To view this whole video, sign in with your Courses account or enroll in your free 7-day trial. Sign In Enroll
Well done!
You have completed Preparing Data for Analysis!
You have completed Preparing Data for Analysis!
Preview
Fixing type and nonsensical issues in the dataset using Pandas.
Binder is no longer available for this project, please follow along by downloading the project files from te downloads tab.
Taking this further
What are some other ways you could tackle cleaning the dataset? I encourage you to think about how you could clean the dataset in fewer steps to challenge yourself. Is there a way functions could be used to help find and/or clean more things at once?
Related Discussions
Have questions about this video? Start a discussion with the community and Treehouse staff.
Sign upRelated Discussions
Have questions about this video? Start a discussion with the community and Treehouse staff.
Sign up
We're almost done.
0:01
Let's tackle the type issues next.
0:02
If we remember, we run data dot info,
you can see our height and
0:06
our weight columns are still objects
instead of numerical columns.
0:10
And if we do pd.to_numeric (data
0:16
[*Height (in)* ] ), and we run it.
0:20
You can see we get an error because
we have some elements that can't be
0:26
converted into a number,
into an integer in this case.
0:31
Because we have 16, and
then we have this inches symbol.
0:35
So let's run a contains and try and
0:38
find all instances where some sort of
text is being used in this column.
0:40
So I'm gonna scroll up and I'm actually
just going to copy what we used earlier,
0:46
I'm gonna paste it over the top of this
because we know this is not going to work.
0:50
Change this to height, inches.
0:55
We're gonna do some regex this time.
0:59
We're gonna check for a through z,
capital letters A through Z,
1:03
and then we're gonna do this slash,
and we only need one for
1:09
the inches symbol,
that double quotation mark.
1:14
We needed to change regex to true and
then we also need to do na equals false.
1:20
And there we go, we got four errors.
1:28
I'm going to have you pause me,
try and fix these on your own, and
1:31
unpause me when you're
ready to see the solution.
1:35
Here we go.
1:38
How did you do?
1:40
Here's an option.
1:42
I did each one on its own line
instead of splitting it up,
1:44
if I scroll back up to
where we did formatting,
1:46
instead of splitting it up into
two lines like we did earlier.
1:49
So to do it on one line,
you still do your data.location, and
1:52
then you pass in the ID number.
1:56
And then you say we just want to
grab this column's information.
1:59
And then we want to set it equal to
the correct format, the correct type.
2:04
Perfect, and
2:11
then you can see I ran it one more time
just to make sure we fixed all of them.
2:11
And then now we can do data, oops,
2:15
"Heights (in)" = pd.to_numeric,
2:20
and then pass in once again.
2:27
And then run it, no errors, perfect.
2:36
And then we could do
a check with data.info.
2:39
And you can see the Height
column is now an integer.
2:42
Amazing, let's do the exact same thing for
our Weight column.
2:45
So I'm going to scroll up.
2:49
I'm not even gonna run the pd.to_numeric.
2:51
I'm just gonna assume that we also
have some text inside of that column.
2:56
So let's do, so
we only need to change this.
3:00
We've that, and
we're gonna keep the same regex,
3:06
there probably aren't any inches symbols
but it's not gonna hurt us to leave it.
3:09
So let's run it and see what we got.
3:15
Looks like we got these two.
3:17
Let's go ahead and fix those.
3:19
We'll do it on the one line method.
3:21
So location 24, and
we want the Weight column and
3:23
we wanna set it equal to 143.30, and
3:28
I'm just gonna do these
both in the same cell.
3:33
And we need 78, Weight (lbs),
3:39
and set that equal to 66.10.
3:45
Perfect.
3:52
I'm gonna run this again below
just to make sure, and amazing.
3:53
Now, I go over the top of this one and
3:59
let's do the same thing, data,
4:03
weight (lbs) = pd.to_numeric.
4:07
I'm really having trouble spelling today,
there we go.
4:12
Data, Weight(lbs).
4:15
And if I run it,
I'm still getting an error.
4:22
It looks like we have a weight that
was over 1,000 and so it added in this
4:26
comma here, so we need to remove that
comma so we can complete the conversion.
4:31
So I'm gonna come up here to our rejects
or regex, whichever one you say.
4:37
And I'm gonna get rid of all of this
cuz we only need to check for a comma.
4:42
Now when I run it, you can see we
got it right there at Snorlax.
4:48
So data.loc, 145,
we want the Weight column.
4:52
We wanna set this equal to 1014.10.
5:01
Enter.
5:08
Wonderful, and I'm actually,
I'm gonna scroll up here.
5:09
I'm gonna copy this one just
to save us some typing.
5:12
And now we can try converting our Weight,
To numeric.
5:17
Awesome, once again, we can do data.info.
5:32
And that one's a float as
it should be because all of
5:35
the numbers have two decimal places.
5:39
So now our Height and
Weight are both numeric columns.
5:43
Fantastic.
5:49
Finally, we can tackle the last item,
nonsensical data.
5:51
Nonsensical.
5:57
Now in our data set,
we have Weight and Height columns and
6:01
all of that information should
be positive, not negative.
6:05
We should never have negative weight or
negative height.
6:09
So we can run data, where
6:13
data height are inches.
6:18
And we can do less than zero.
6:27
And it looks like we don't have any
in height, so let's try weight.
6:32
Weight (lbs), looks like we have two
here inside of our Weight column.
6:36
This is one where we can go to Google and
we can check these values to see if this
6:44
weight is incorrect or
if it's just accidentally negative.
6:49
Hopefully I spelled that
close enough Pokedex.
7:00
I scroll down, 105.8.
7:04
Alakazam, 105.8, so
it's accidentally negative.
7:08
And just to save us some time,
this one is also accidentally negative.
7:12
So what we can do is we
can run absolute value,
7:16
because that will switch all
negatives into positives.
7:19
So it looks like this,
data, where, oops, data,
7:24
height, no, weight,
we're on the weight column.
7:28
(lbs) = data
7:37
Weight (lbs).abs.
7:42
Now we'll run the absolute value for
7:51
each item in the row, or in the column.
7:54
So if I run this again,
you can see now I get nothing.
7:59
Everything has been made positive.
8:02
Fantastic work.
8:06
To run a final check, because our
dataset's a little bit smaller,
8:08
we can do pd.set_option.
8:13
Display.max_rows, none.
8:18
Now if I run data,
8:28
it's going to show me every single row in
our dataset instead of just the first few.
8:29
Because our dataset's smaller,
8:35
this is something that you
can do to check your work.
8:36
And because we're practicing, obviously,
if this was a larger dataset,
8:39
this wouldn't be as reasonable.
8:43
You tackled cleaning
a dataset using pandas.
8:45
Before you go, I want to remind you to
use your Googling skills when it comes to
8:49
solving any other errors you come across,
or even for
8:53
looking up different ways to solve
the errors that we already did.
8:56
For instance, let me show you an example
of a Google search that I did.
9:00
Change negative to positive.
9:06
We don't want in Excel we want pandas.
9:10
And here we have turning column of
mixed positive from stackoverflow, and
9:17
if I click into it, you can see they have
mixed positive and negative numbers.
9:21
And it looks like everyone's saying
to give absolute value a try.
9:27
And you can see here, the first answer
right there is how to use absolute value.
9:31
And if I'm like,
I'm not sure what that is, and
9:36
I wanna read more about it,
I can do pandas abs.
9:39
It'll pull up our pandas documentation,
and
9:44
I can read more about what that does and
see some more examples.
9:47
See how they had this series,
they ran abs, and now all positive.
9:53
This is just to remind you that no one
has memorized every possible solution to
9:59
cleaning a dataset.
10:04
We all google when we get stuck or
forget how to tackle an error,
10:06
so don't be afraid to pop into Google and
start searching.
10:10
In fact, knowing how to search for
a solution is a huge skill for
10:14
any developer or analyst.
10:18
You need to sign up for Treehouse in order to download course files.
Sign upYou need to sign up for Treehouse in order to set up Workspace
Sign up