Parallel Go command line program to read GraphQL and process private team members

code golang graphql microsoft teams parallel programming

I was approached by our Sharepoint staff to help with a problem. We needed to find groups in our Microsoft Teams environment that had private channels in them. From their research, even admins can't run a simple search to find private channels unless they were a member of all teams. My GoogleFu revealed the same. Our admins said that Microsoft's suggestion was to use GraphQL and I'm the main GraphQL experienced person in our department they know of.

Wanting to do this in a repeatable method without creating a web site that needed to be maintained, authorised, etc., I thought I'd create a quick command line tool to do the querying for them. A coworker had shown me command line tools he'd written using Golang's Cobra and Viper modules. I'd used that to rewrite my PHP blog generator into a Golang tool but hit some roadblocks with the markdown parsers and my own decisions on HTML templates. But it was the perfect tool for this.

Initial Process Map

  1. User signs in. An admin user so they can access everything.
  2. Run through the GraphQL endpoint for Groups, getting all the groups. Paginated, so repeated calls required
  3. Iterate through this list and for each Group find the Channels that are Private
  4. Get all the information required on these Private teams and print them out
  5. Done

Build 1

Built an initial project using Cobra

  • src
    • cmd
      • lookup.go
      • root.go
    • sso.go
    • go.mod
    • go.sum
    • main.go

Main.go holds the "Just start cmd" stuff for Cobra
src\cmd\root.go holds the Cobra overall stuff for a command line tool and config variables
src\cmd\lookup.go holds the GraphQL integrations for performing the Lookup command
src\cmd\sso.go holds the Single Signon component

To sign in as a user (and restrict access) I created a new application in Azure Portal, given the client/ secret as config to the code when compiled. I'd written this little library previously and I am amused by how it works.

  1. Call promptForLogin() to start the login process
  2. Starts an internal webserver that responds to a specific port
  3. Add a http.HandleFunc that responds to the /login endpoint to process the returned code from the Azure OAuth
  4. Start the server
  5. Open the OAuth login window in the browser, pointing at localhost:port as the post-auth URL. Do this by looking at the runtime.GOOS variable
    1. linux ? use xdg-open
    2. windows ? use rundll32 url.dll,FileProtocolHandler
    3. darwin ? use open
    4. Otherwise I've no idea how open a brower here 1
  6. Wait

Once the user logs in, the browser is directed to the internal server Go is running. So it gets the code, can perform SSO, and informs the user they can close the window. Handy

Now I've got a token, I use a collection of GraphQL URLS to get my information

  • /beta/groups has the list of all groups. Pass in the query parameter of resourceProvisioningOptions/Any(x:x eq 'Team') to get teams.
    • Optionally the user can supply a name to prefix match, this is added to the query parameter as and startsWith(displayName, '%s')
  • /beta/groups/%s/owners to get the group owners
  • /beta/teams/%s/channels to get all channels. Pass in the query parameter of filter=membershipType eq 'private' to get only private channels
  • /beta/teams/%s/channels/%s/members to get all members of the specified channel. Pass in the query parameter of filter=roles/Any(x:x 'owner') to get owners.

Plugging all this together, the code runs! But... the data returned is incorrect. It can still only get members of teams I'm part of. Investigating things I found the delegated permissions to the scopes don't work even for admins looking for channel members they're not team members of. So I get application permissions and now it works.

Very. Slowly.

Have to go through all the queries to get the list of all possible teams, then go through each of those to find all possible channels. All serial processing.

Good thing Go has parallel processing built in.

Parallel Process Map

  1. Set up a series of internal channels
    1. A TeamsToProcess channel that's filled with the found teams
    2. A WriteToFile channel that's filled with the things to write to screen or file
  2. User signs in.
  3. Using go to run in parallel: Run through the GraphQL endpoint for Groups, getting all the teams. Paginated, so repeated calls required.
    • After each page is returned, add all found entries to the TeamsToProcess channel
  4. Using go to run in parallel: using runtime.NumCPU to start a number of workers: Pull the next entry off the TeamsToProcess channel and look that one up
    • Get all the information required on these Private teams and add that to the WriteToFile channel
  5. Using go to run in parallel: Open a file and start writing anything adding to WriteToChannel to that file. STDOUT is the file if no file is specified
  6. Done

Build 2

 1		teamsToProcess := make(chan TeamValue)
 2		writeToFile := make(chan string)
 3		m := sync.Mutex{}
 4
 5		wg := &sync.WaitGroup{}
 6		// IN PARALLEL
 7		// Get all the matching teams
 8		go getAllTeams(teamsToProcess, wg)
 9		// Process each found team
10		for i := 0; i < runtime.NumCPU(); i++ {
11			wg.Add(1)
12			go func(teamsToProcess chan TeamValue, writeToFile chan string, wg *sync.WaitGroup) {
13				defer wg.Done()
14				// Get the Team owner
15				for team := range teamsToProcess {
16                    // Processing goes here
17					if ShowProgress {
18						index := SpinInt % 4
19						fmt.Printf("\r%d%c", SpinInt, []rune(`-\|/`)[index:index+1])
20					}
21					m.Lock()
22					SpinInt++
23					m.Unlock()
24				}
25			}(teamsToProcess, writeToFile, wg)
26		}
27		// Tell the writer to write what we find
28		go func(writeToFile chan string, wg *sync.WaitGroup) {
29			var f io.Writer
30			var err error
31			if len(OutputFile) == 0 {
32				f = os.Stdout
33			} else {
34				f, err = os.Create(OutputFile)
35				if err == nil {
36				} else {
37					fmt.Printf("Could not open %s for writing\n", OutputFile)
38					os.Exit(4)
39				}
40			}
41			outputWriter := bufio.NewWriter(f)
42			outputWriter.WriteString("[CSV File Headers]\n")
43			outputWriter.Flush()
44			for s := range writeToFile {
45				outputWriter.WriteString(s)
46				outputWriter.Flush()
47			}
48		}(writeToFile, wg)
49
50		wg.Wait()
51		// Finished processing, now we wait for the file to finish writing

Wanted to call fmt.Printf("\r%d%c", SpinInt, []rune("-\|/")[index:index+1]) out. It uses "\r" to reset the cursor to the start of the line in the terminal, then it writes out the current count of process, and has a nice spinning text throbber to show it hasn't crashed. Really nice.

Run this one and it works a TONNE Faster. Mucho Parallelo.

  • New Application in portal.azure.com
    • Permissions must be Application level to be able to get members of teams you're not part of, even if logged in as an administrator
    • Permissions must have an administrator grant for the application to use
      • Sites.Read.All - Get sites
      • Group.Read.All - Get groups
      • ChannelMember.Read.All - Get members of channels
      • User.Read.All - Get information on the owners beyond their ID
    • Authorisation URL is a localhost with a specific port
  • New Cobra app
    • Command: Lookup
    • Flags:
      • -p: Show progress
      • -o [file]: Save the output to this file. Otherwise output it to the screen
      • -g [group]: Look only for groups that start with this prefix. Otherwise get all groups
  • New Cobra command Lookup
    • Login via OAUTH
      • Run an internal web server w' specific port to get the response
    • Create two channels
      • One to add found Teams to
      • One to add output to
    • In Parallel:
      • Start going through the paginated GraphQL endpoint to get all groups (Teams)
        • For each returned page, add to found Teams channel
      • Create a number of parallel processors to process the found Teams using the GraphQL endpoints to get channels and members
        • For each private channel, add to the Output channel
      • Start watching the Output channel, and write to the appropriate file endpoint
  • Done!

I'd love some feedback.

Update

I've been told of a go library dedicated to opening browser windows. Going to investigate that and possibly get more options for where to run the tool.

  1. Really tempted to include Lynx in this instance. ↩︎

Professor von Explaino