Translation leaderboards are meaningless; here’s how to fix them

Part of my series of answers to frequently asked questions about community participation in localization or translation projects.

I’ve often considered creating a FAQ on this site covering answers to common questions about community-driven localization. Without fail, the most common question I get is, “But if they’re all volunteers, how do you ensure quality?” I can get into the issues with the underlying assumption in that question later. This time around, I’m going to address one of the more common quality control mechanisms for (dare I say it) crowdsourced localization and translation projects: the leaderboard.

Continue reading Translation leaderboards are meaningless; here’s how to fix them

Some questions I’m asking about community localization workshops

Localization workshop with Mozilla LATAM community in Lima, Peru, 2015. Look at that beard, yikes!

I’ve planned a lot of community localization (l10n) workshops all over the world. Let’s just say that after six years at Mozilla, it’s approximating 50 at least. They’ve all taken different shapes over the years and as I begin working with local community members and the Mozilla l10n-drivers to plan the last three community l10n workshops for year, there are a number of questions I’m asking myself about these workshops and beyond.

Why do we organize these workshops?

Organizing these things are hard work. Often it’s a lot of fun, but it’s time-consuming and full of variables. Here are some of the reasons we do these:

  • In an open source community, where the people involved are generally self-organized and the expectation of a bottom-up style of management, it’s crucial for paid staff to travel and meet with volunteer staff (Stormy Peters has some good thoughts on this). This humanizes the people involved, it unifies them all under the organization’s common goals, and gives them the opportunity to voice their ideas, concerns, and opinions to a real person.
  • To train and give mentorship to community and give community to train and give mentorship to one another.
  • To dedicate time to projects that support the community but wouldn’t be a typical part of their contributions (e.g., creating translation style guides).
Are these workshops relevant to those attending?

I’m afraid the format of our workshops has grown stale and static. There are so many variables to consider when planning the format of a workshop. The community’s level of maturity, personality differences, cultural differences, varied skill sets, and tenure in the project. The more communities you bring together for a workshop, the more complex it can become. With all these variables, a static workshop format is doomed to failure. But what does a dynamic format look like and how do we know what to adapt to the needs of the communities present?

I’m mulling over several ideas about format that we may try for our last three workshops in 2017.

  • Mozilla localization unconference — everyone comes with topics they want to discuss or lead a discussion on, the agenda is defined together in the morning and the rest of the weekend is spent moving down the list. This facilitates the need for a dynamic format, as no two workshops would look alike and would largely be planned by those present.
  • Problem solving — the weekend is spent brainstorming and offering solutions to a problem that either a high number of community’s are facing or a problem that one community is facing.
  • Rotating topic — rather than organizing workshops to cover all of the regions of the world, workshops would be organized around specific topics or deliverables. The members of the global community that can most successfully contribute to those would be invited. No two workshops would be the same.
  • Work week with l10n-drivers — with some advanced planning, all l10n-drivers attend the workshop and go about treating the workshop as a team work week. Community will have their pick of contributing to the different projects the l10n-drivers are working on and would see localization from a number of different angles.
What is the right size for a community workshop?

I’ve been in and organized workshops involving one community (around 5 people) and workshops involving fifteen communities (~60 people). I can’t definitively say that one is better than another. The great thing about large workshops is that everyone is introduced to a significant amount of diversity and expands their worldview as a result. It’s hard for the l10n-drivers to have significant personal interactions with each localizer present though. In smaller workshops, you miss the diversity and potential for learning, but the l10n-drivers get more personal time with the localizers and there’s more freedom to dig deep into specific issues that uniquely affect that community.

This year we’ve been organizing large workshops. I’m estimating that we’ll send about seventy invitations to our Berlin workshop. One piece of feedback we’ve gotten consistently is that the Mozilla community wants to see more diversity at these workshops. Organizing larger (but fewer) workshops could help meet that need. But at what point does the workshop become a conference?

How do we determine if our workshops over the course of the year were successful?

We hold ourselves accountable for making sure that the time, effort, and money spent on these workshops yields specific results to measure whether or not the events were successful. This success criteria changes from year to year. It started out being number of strings translated at the event (which was a really bad metric) and has evolved into whether the event has helped communities to meet their own goals. If the format becomes more dynamic, how does this change our success criteria? I don’t have an answer for that yet.

Are workshops the right tool for accomplishing our purpose?

I started this by asking why we organize these, and on paper it seems that, yes, workshops are a great tool for accomplishing those purposes. I suppose this question is better stated as, “are workshops the only right tool for accomplishing our purposes?” I don’t have an answer for this yet. The Web breaks down geographic barriers for collaborating, and while I see the value in moving away from our computers and interacting face-to-face, we’re getting closer to a world where the idea of being face-to-face doesn’t have to mean traveling across land and sea. I’m not sure that we’re really taking advantage of what the Web has to offer to break those geographic barriers.

I’m going to keep asking these questions and I think they’re general enough that the answers will be as dynamic as I hope our workshops become. Get in touch with me if you have thoughts on how to answer any of these questions.

Professor Beatty

About 8 years ago I was an undergraduate student at Brigham Young University (BYU) in Provo, UT. I was taking the LING 480 course on translation technology and my world was opened to this new realm of technology and an industry that I had really not known to exist before. It was enthralling! At the time, I distinctly remember daydreaming into the distant future, toying with the strange idea that I could one day teach this course. Call it a premonition…

Continue reading Professor Beatty

Evaluating localization tools: formulating the right criteria

It can be very difficult to evaluate a localization tool and decide if it fits within your localization process and delivers, not only as advertised, but also according to your localization strategy’s specific needs. Your tool evaluation criteria may depend on where localization takes place in your development cycle, if you’re a freelance localizer or selecting enterprise localization solutions, and how well the tool manages your (or your client’s) language assets.

Continue reading Evaluating localization tools: formulating the right criteria

Agile localization: l10n PMs, step aside

As we’ve read in Dunne’s book, one of the roles of the project manager (PM) is to faciliate all parties in their ability to efficiently complete their tasks. This often translates to what many of my colleagues have already labeled as, “File Pusher”-syndrome. I understand where this perception comes from; the PM is most visible it seems when they’re facilitating a project’s execution. Now that the world is moving to the Agile methodology, localization PMs need to find a leaner way to work and a less intrusive way of facilitating project execution.Why not turn to automation to fix this problem? Smarling already did, and it seems to be working well for them.

Continue reading Agile localization: l10n PMs, step aside

“What do you do for a living?”

My career is non-traditional. By non-traditional I mean that stating my job title does not give anyone the faintest idea of what I actually do (e.g., you say, “Engineer,” everyone gets it). As you can imagine, this has caused me some problems whenever people ask me what I do for a living. I’ve had to practice many 30-second explanations of what I do. Want some examples? Maybe some of you have heard one or two of these at one point:

  • “I help make Firefox available to people all over the world.”
  • “I help global volunteers produce a regionally appropriate version of Firefox for users in their countries.”
  • “Have you heard of localization? It’s the process of taking a product and adapting it to meet the needs of consumers in a particular region of the world. I do that for Firefox.”
  • “I drive a variety of projects that allow Firefox and other Mozilla products to be available in over 90 languages around the world.”
  • “It’s my responsibility to make it easy for global volunteers to adapt Firefox to meet their local needs.”

Continue reading “What do you do for a living?”