- A former Facebook moderator named Sarah Katz described the pressure she was under to review thousands of posts a day from the darkest corners of the platform.
- Katz told Business Insider that she eventually became desensitised to child porn and bestiality because she saw so much of it.
- Her experience raises questions about Facebook’s efforts to stamp out child exploitation on its platform.
- But Katz, 27, said she was proud of her work and urged tech firms to do more to make moderation a career.
- Facebook said it recognised that moderation can be difficult but that employees “play a crucial role in helping create a safe environment for the Facebook community.”
A former Facebook moderator said the pressure to churn through a never-ending pile of disturbing material eventually made her desensitised to child pornography and bestiality.
Sarah Katz, 27, worked as a content reviewer at Facebook’s headquarters in Menlo Park, California, through a third-party contractor, Vertisystem, for eight months in 2016. Her job was simple: figure out whether posts reported to Facebook violated the company’s detailed community standards.
Practically, this meant eyeballing new and potentially horrific material every 10 seconds and making a snap decision about whether it needed to be ditched. Posts that needed reviewing were called “tickets,” and there were about 8,000 every day.
To deal with this onslaught, Facebook had 4,500 moderators like Katz on its books last year, and in May 2017 it announced plans to hire another 3,000 to help it in the fight against the darkest corners of its user output. Facebook is also investing in artificial intelligence to help police posts that break its rules.
Facebook detailed the scale of its problem with prohibited content in a transparency report in May. It said that in the first three months of this year it “took action” on 21 million posts containing nudity and sexual activity and 3.4 million featuring graphic violence. Millions of posts with hate speech, spam, and terrorist content were also removed.
Reviewers have to sign a waiver document about offensive material
Content reviewers begin their Facebook journey by signing a waiver document, which basically acknowledges that they are braced to view the disturbing material. It’s also designed to protect Facebook from any potential legal action.
The one Katz signed says moderators will be exposed to material that “may be offensive to some people,” including pornographic images. It adds that staff members should “promptly notify” Facebook if they “do not wish to continue.”
“Facebook has billions of users, and people don’t know how to use the platform correctly, so there was a lot of pornography, bestiality, graphic violence,” Katz told Business Insider. “There was a lot of content that you might not expect to see shared on Facebook.”
Katz worked in an open-plan office in Menlo Park, where free snacks flowed and there was reasonable camaraderie among her colleagues. They would set to work on their queue of posts for review, and when in full flow, Katz would make decisions within seconds, she said.
If ticket targets were not met, there would be consequences. Failing to hit a goal “once or twice” would result in a warning, Katz said; more than three times, “you would probably get let go.” Katz said that she never witnessed this but that it was informally known among staff members.
“It’s kind of a monotonous job after a while,” she said. “You definitely grow desensitised to some of the graphic material because you see so much of it. A lot of the content tends to recirculate.”
A sinister image that kept resurfacing
Katz said there was a particularly sinister photo and video that popped up repeatedly in her queue.
She said it featured two children between 9 and 12 years old standing facing each other, wearing nothing below the waist, and touching each other. It was clear, Katz said, that there was someone behind the camera telling them what to do.
“It would go away and come back. It would appear at multiple times of the day,” she said. “Each time the user location would be different – one day shared from Pakistan, another day the US. It’s kind of hard to track down the initial source.”
At the time, Katz said, she was not asked to report the accounts sharing the material – something she said “disturbed” her.
“If the user’s account was less than 30 days old, we would deactivate the account as a fake account,” she said. “If the account was older than 30 days, we would simply remove the content and leave the account active.”
Her experience raises questions about the effectiveness of Facebook’s efforts to tackle child exploitation on its platform.
The company signed a deal with Microsoft in 2011 to use its PhotoDNA technology, designed to scan all images on Facebook and Instagram, flag known child porn, and prevent it from being reuploaded. Furthermore, Facebook moderators are trained to recognise and escalate such content internally when they see it.
Facebook told the New York Post in 2012 that it reported all instances of child exploitation to the US’s National Center for Missing and Exploited Children.
“We have zero tolerance for child pornography being uploaded onto Facebook and are extremely aggressive in preventing and removing child exploitative content,” a company spokesman said at the time.
Katz said she was not aware that any of this was in place in 2016.
“Facebook might have a policy [now] where you’re supposed to report it, but back then they didn’t,” she said.
Facebook declined to comment on the discrepancy between Katz’s account and its stated policies.
The nuance of human perspective
Katz is now a cybersecurity analyst for the cloud-computing firm ServiceNow and has written a sci-fi novel, “Apex Five,” that draws inspiration from her time at Facebook. She is broadly upbeat about her Facebook experience, arguing that the downsides of the job were outweighed by the sense that she was protecting users.
“There needs to be many human eyes to do the job, and it cannot all be allocated to artificial intelligence, no matter how much folks say,” she said. “We will always be that nuance of human perspective. AI would help to track all that content from billions of users. The human element has to be taken care of, so we keep doing a good job.”
Katz urged tech firms to do more to make moderation a career rather than a short-term job.
“It behooves not only Facebook but social-media platforms in general to hire content moderators on a full-time basis because provides much more incentive,” she said. “It really incentivises us to do a stellar job and make it something we want to stick with, rather than winging it, holding out for something better.”
A Facebook spokeswoman said: “Our global team of content reviewers play a crucial role in helping create a safe environment for the Facebook community, which is our number one priority. We recognise that this work can often be difficult, so we have wellness and psychological support in place for all our staff.
“It’s a really big job with more than 2 billion people on Facebook, so we recently added 3,000 people to our existing 4,500-strong community operations team around the world to review the millions of reports we get every week.”