Handle invalid Unicode characters during import CSV upload

[orenyk]

Currently, if you pass in a CSV file with an invalid unicode character during import (see this article for more details) it just throws an exception (ArgumentError (invalid byte sequence in UTF-8)) and sends back a 500 error, so the user has no idea what's going on. We should either sanitize our inputs (see the previously linked article) or rescue the exception.

[esoterik]

I'm having trouble replicating this issue -- I'm uploading a CSV with several invalid unicode characters, but they're not being processed properly. For instance, having Amazon Basics 60-inch \255 in the CSV simply turns into "Amazon Basics 60-inch \255", with no errors (gsub works and everything).

[orenyk]

I don't know if that will work - the CSV file itself is probably unicode-encoded, so I'm not sure that \255 is read by Ruby as that unicode character, as compared to that set of four characters. I'm not sure exactly how you'd go about entering one of those characters; I'm sure there's a way to enter a custom character in vim but I wouldn't know it off the top of my head. Hope that helps!

P.S. While looking into this I remembered the bug that prompted this; a user tried to upload a CSV file with spaces between the headers (e.g. name, max_per_user, max_checkout_length,... vs name,max_per_user,max_checkout_length,...). Since this seems like a common user error, adding this to our CSV validations would be awesome 😄!

[esoterik]

oh okay that makes sense.

I'll add that to the validations!

[esoterik]

for posterity: here's how to insert characters by value in vim.