Onwards and Upwards

The journey called SocialCops started six years ago. Two 21-year-olds, fresh out of college, arrived in New Delhi with a dream to solve real-world issues by harnessing the power of data.

And soon, an amazing team of dreamers joined us on the journey. When we moved into our current office, we wrote up a dream wall. Our wall had dreams like “Power a national-level policy decision”, “Prime Minister using our platform” and “Global expansion”.

Our journey at SocialCops is a powerful example of how small teams can achieve large-scale impact—we were a lean team of 50. It reminds us that meritocracy exists in our world (we started as two 21-year-olds with no political connections), and most importantly of how with the right team, hard work and passion, you can achieve just about anything.

Today, many of those initial dreams have come true. As data intelligence partners for the historic national program, Ujjwala Yojana, we touched the lives of 80 million women below the poverty line. We built India’s landmark DISHA where data from 41 ministries and 22 sectors sits in one place—live, dynamic and updated down to the last village in India, launched by the PM himself! And we partnered with the United Nations to help countries globally track progress towards the Sustainable Development Goals.

While we achieved many breakthroughs in the world of data, the ones we are most proud of are those that we achieved with our team… the human ones! Our biggest learnings in the past 6 years have been around people, teams and what it takes to make them successful.

And, data teams are the most complicated of them all 😖

They are more diverse than possibly any other type of team in the world. To solve a data problem successfully, you need an entire team of individuals with complementary skills to come together—from engineers solving for scale to scientists building models, business users who best understand the problem, machine learning engineers, data analysts and so on.

While diversity was our biggest strength, it was also our biggest barrier to scale. Every single person on our team had a different way of approaching a problem, different tools, skill sets, DNA… they basically had nothing in common except a common goal!

Soon, we started seeing cracks—mostly due to small issues, not large ones. 

  • An engineer pushed the wrong data file to production.
  • An analyst used the wrong variable in the analysis.
  • An API spec was changed by a client without telling us.
  • A data scientist left and rerunning the model they built was crazy.
  • Our ML engineers struggled to explain how a model worked to the business users. 
  • Business users complained that the data team was a black box.
  • Analysts complained the data pipeline was a black box.
  • Engineers complained that the client problem statement was not properly communicated to them.

Even so, we pushed through and completed many successful projects by thinking on our feet and implementing quick fixes. But we realized that there was no way we could scale like this. So… 

We began to build small tools to help our team collaborate effectively.

These started with small fixes—like opening up the data pipeline visually (we used Airflow) so that analysts could troubleshoot failures and to help build trust among the team when something failed. We started to automate basic things by setting up alerts and notifications. Then we solved for data discovery and access. Then knowledge management. And so on.

They say small steps add up to something big. By the end of 2017, our team was performing at 2x the quality, as we needed far fewer iterations to push a final product… in 50 percent of the time, thanks to the power of automation… and with 1/3rd the resources, given reduced engineering time.

Powered by these tools, our team went on to implement some landmark projects. As we inched closer to achieving our dreams, we began to envision our future. And we began to think…

What if we shared our tools with data teams around the world?

Could we make them achieve their goals faster, just like we had done with our own team? Around the same time, data teams were becoming more mainstream in companies and we realized we had the opportunity to multiply our impact on the world. 

We believe that every human achievement in the next century from humankind landing on Mars to finding a cure for cancer (hopefully!) will be powered by an amazing data team. By sharing our tools with these teams, we could help them accomplish incredible things, and in turn speed up human achievement and progress.

Introducing Atlan—a home for data teams.

Where you can bring together diverse data, tools and people to create a frictionless collaboration experience. Think of us as the glue—the collaboration layer where you can work with data and all your tools.

We’ve spent the past 18 months in stealth, working very closely with our beta customers to validate our ideas and ensure that we’re adding value to these teams. And we’re already seeing signs of success.

  • We started powering teams in 50+ countries and growing with supersonic speed 🌏
  • We are now loved by amazing data teams in over 200 organizations! 👫
  • Our hearts are warmed by users saying things like, “Your product saves me 2 hours a day so now I finally have time to learn Python!” ❤
  • We raised funding of $2.5 million 🚀

Do check out www.atlan.com to learn more.

What’s next?

Going forward, SocialCops will continue as a data for social good community and we hope to open-source our learnings to help the larger community. As we scale Atlan, going forward, our team will devote their time and attention to building products that help data teams do their lives’ best work.

Finally, our first social media post when we launched SocialCops six years ago was a saying that inspired us, “A small group of passionate people can change the world… indeed it’s the only thing that ever has.” Our journey at SocialCops is proof of how true that statement can be!

We would like to thank each and every one of you for being a part of our story and making it so beautiful.

Signing off, 

Team SocialCops ❤

The post Onwards and Upwards: Chapter 2 for SocialCops appeared first on SocialCops.