Our data engineer insists in lowercasing everything and removing some other formatting like new lines on free text fields.
They say it’s “better for elastic search”.
To me that makes no sense and loses information that can’t be added back. But I couldn’t really convince them otherwise. So far no real problem has come out of it but it makes for a worse experience for the user. Like company names that are acronyms show up as all lowercase. (ibm, llc, etc.) or free text fields that we miss when the user wrote in caps or added paragraphs.
What are your thoughts on this?
Disclaimer, I’m not a data engineer. Just a PM from a data related product.
We build market analytics/reports out of the data from elastic search.
Thank you for your suggestion. I’ll address this with them to see if I can get a better understanding of the reasoning behind it.
We don’t have access to all the past data, most yes. But a lot no.