Can social media scraping find you the right person for the job?

When it comes to easing the burden on recruiters, artificial intelligence (AI) is revolutionizing the way that we work.

From tools that scan resumes to chatbots that field questions from candidates, new technologies are infiltrating HR processes to remove some of the legwork of finding the right person for the job.

The recruitment process has always involved extraordinary levels of data. From detailed resumes to complex candidate assessments, traditional hiring methods required individual recruiters to gather and manually process a significant volume of information.

Advocates of AI argue that we can remove the more time-consuming aspects of the process and save recruiters from wading through reams of complicated data on candidates, creating a more efficient and reliable method of evaluating applications.

But what many applicants might not realize is that AI isn’t just processing their data but also gathering it – and that this might start before they even apply for a job.

What is social media scraping?

Data scraping is the process of extracting publicly available data from a website for analysis.

The technology is widely used for gathering business intelligence — a company could, for example, scrape information from thousands of competitors on pricing in order to evaluate where they sit in the market or pull data from social media in order to understand how customers are talking about their product online.

In recruitment, it has long been accepted that hiring managers might sneak a look at a candidate’s Facebook page, with 71% of recruiters saying that social media decreased the amount of time that it took them to fill positions.

Social media scraping takes scrolling through a potential employee’s online profile to another level by gathering significant amounts of data and running it through an algorithm.

Utilizing AI, it analyzes the candidate’s perceived traits against measures of suitability for the role. Everything from recent likes to the language of posts can be used to assess existing applicants, or even to target job advertisements and recruitment efforts toward users that might be a fit for an open position.

“An individual’s public posts can reveal quite a bit about that person and how he or she may fit into an organization’s recruitment objectives,” explains Attila Tomaschek, a data privacy expert at ProPrivacy.

“Instead of taking the time to manually go through and look at a candidate’s social media posts individually, the hiring manager can use a program to scrape data and run it through an algorithm that automatically analyzes the candidate’s personality traits and potential fit at a company.”

The surprising benefits of social media scraping

Dr Tomas Chamorro-Premuzic is a professor of business psychology at University College London and Columbia University and the author of The Talent Delusion: Why Data, Not Intuition is the Key to Unlocking Human Potential.

He believes some people balk at the idea of recruiters using social media to assess candidates, not trusting machines to make important decisions. But Chamorro-Premuzic argues that rather than introducing errors into the process, as many candidates suspect, AI can actually help to remove biases and serve the applicant in finding roles that they are better suited to.

“There’s no question to me that technology can and should be used to match people to jobs,” he says. “Companies can be really smart with using social media scraping to find pre-hire candidates and target them with adverts for jobs. Facebook has already sold your data to better target you, and I would argue that most people would prefer to be targeted for a better job than a pair of sneakers.”

Chamorro-Premuzic also believes that traditional recruitment tools leave a significant gap between the skills needed to excel in a selection process and the skills needed to perform well when in a role. By scraping a candidate’s social media profile he argues that recruiters can avoid making hiring decisions that could cost the company, while also decreasing discrimination against candidates.

“Having a human in the loop is the best way to perpetuate bias,” he says. “There are already explicit tools for matching people to jobs which seem less controversial because the data is just passed onto human recruiters, who then decide who is the best fit. But those humans are mentally scraping the profiles, and using conscious and unconscious biases to make a decision.”

A legal and ethical grey area

Data scraping might seem like a revolutionary way to dig deep into a candidate’s fit for a role, but recent years have also seen a rise in concerns around privacy that could hamper its adoption.

The introduction of GDPR in Europe has left recruiters wondering whether social media scraping is ethical – or indeed still legal. But the boundaries around how data can be gathered and collected remain blurred.

“Even though the data being scraped is technically public information being posted online by the user there are still a number of privacy concerns,” says Tomaschek.

“Analysis of a group of innocent and seemingly unrelated posts can instantly generate a detailed and intimate representation of a person’s daily habits, interests, religious beliefs, and more. This isn’t necessarily the type of detailed profiling a person would want to be conducted on them, even by an organization at which they are actively seeking employment.”

In Europe, GDPR requires the informed consent of a subject when it comes to how their data is collected and used. This won’t necessarily apply to social media, where data is often publicly available and users will likely have agreed that their information can be shared when creating an account, yet some social media platforms are now discouraging the practice.

Protections in the US are less strict, but some privacy experts argue that data scraping should still be used very carefully in order to prevent accusations of discriminatory behavior.

“Screening potential candidates on social media may put employers at risk of discrimination complaints, particularly if the information they discover online is related to characteristics that are protected,” says Kate Palmer, HR advice director at global law consultancy firm Peninsula. “Candidates’ social media accounts can seem like a helpful tool that you have available, but you should use them with caution.”

What’s next for social media scraping in recruitment?

Recruiters eager to source the most detailed information about a candidate might find themselves unsure about whether social media scraping can help.

The rules around data collection are still being written, but for now, experts advise being upfront about how data is used and carefully considering the ethical implications of mining an applicant or potential employee’s online life.

“It’s important that we educate people about how to use data in an ethical way,” says Chamorro-Premuzic. “Does the person opt in? Is there informed consent? Will you preserve their anonymity as needed? Will you keep their data secure and safe? And, fundamentally, is there a benefit to the user?”

While social media scraping is still permitted it’s important to exercise care and due diligence and to respect applicants’ data. Recruiters also need to remember that while applicants might only show the best parts of themselves in an interview, social media also isn’t always the best measure of a person.

“No one is themselves when they go to a job interview, but then nobody is really themselves when they create a social media profile,” says Chamorro-Premuzic. “Technology is neither good, nor bad, nor neutral. It’s the application of it that’s important, and we have a role to make people’s understanding of it more nuanced.”