[cs615asa] HW#N: Music is Just Wiggly Air

Mark Freeman mfreema1 at stevens.edu
Wed Feb 26 18:17:33 EST 2020


Hi all,

Yesterday, I attended the “Music is Just Wiggly Air” meetup at Spotify,
which is a series of talks focused around scaling data processing for audio
analysis and research.  I decided to attend because data processing,
particularly in large-scale distributed systems, is an interest of mine
(also, I could not resist the name).

I learned that for a company like Spotify, handling data processing jobs
(commonly called transforms) is quite a bit about performance, but also
usability.  Audio researchers (the people responsible for the algorithms
that extract information from and categorize audio) are not always computer
scientists by education.  Oftentimes they have an understanding of Python,
but not too much past that.

This is where system administration comes in.  Spotify uses the ideas of
system administration to orchestrate and automate dependency management,
containerization, and worker scheduling so that their researchers can focus
as much as possible on the actual algorithms that they are developing.

The particular tool they use is called Klio.  It is an internal Scala API
that coordinates application of a directed acyclic graph of transforms with
minimal tuning.  Unfortunately, it is not yet open source, but its sister
project, Scio, is: https://github.com/spotify/scio

Overall a pretty neat experience!  Their offices were very cool, lots of
audio stuff (imagine that).

Side note:  I also got to meet an engineer from Bonobos (the clothing
company) at the event who told me a little bit about what it’s like to do
technology at a company focused primarily on other things.

He explained that it pushes you to do quite a bit more end to end things
than you would otherwise.  Everyone is a developer, but they are also
pseudo system administrators, pseudo database administrators, etc.  As he
put it, “somebody has to keep the lights on.” If you need security
credentials for something, you just lean over and ask for them.

Not saying this is not how it should be done, after all they are a team of
only 30 people, but I think it goes to show how wildly different system
administration can be depending on your context.  It does not fit into an
exact definition as we have already seen.

Thanks!

Mark Freeman
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.stevens.edu/pipermail/cs615asa/attachments/20200226/89d398da/attachment.html>


More information about the cs615asa mailing list