Ggtcf

When I paginate through next_cursor, I get a different number of channels every time I hit the URL through NodeJS function.

Once the total channels fetched is 7488 another time it is 300. It differs every time I run the program.

URL : https://slack.com/api/conversations.list?types=public_channel&cursor=cursor=&exclude_archived=true&token=token

The issue is due to the rate limit of slack. conversation.list comes under tier 2 rate limit. That is only maximum 20 requests per minute.

function fetchData(){
 getResponse(url);
 function getResponse(url) 

 let tempData = '';

 https.get(url, (resp) => 


 resp.on('data', (chunk) => 
 tempData += chunk;
 );

 resp.on('end', () => 
 try
 tempData = JSON.parse(tempData);
 if(tempData.channels)
 resultData.push(tempData.channels);
 

 if (tempData.response_metadata && tempData.response_metadata.next_cursor) 
 if(tempData.response_metadata.next_cursor === '')
 return resultData;
 
 let cursorIndex = url.indexOf('cursor');
 let newUrl = url.slice(0,cursorIndex);
 let token = apiConstants.SLACK_API['ACCESS_TOKEN'];
 let nextCursor = tempData.response_metadata.next_cursor.slice(0,tempData.response_metadata.next_cursor.length-1);
 nextCursor = nextCursor + "%3D";
 newUrl = newUrl + 'cursor=' + nextCursor + '&token='+ token;
 getResponse(newUrl);
 else 

 return resultData;
 
 catch(err) console.log(err); 

One way to do this reliably would be to use the Node Slack SDK's WebClient, which has automatic pagination on cursor-paginated methods (like conversations.list). It also automatically handles rate limits by queuing requests for the amount of time the responses from Slack indicate. Disclaimer: I work at Slack and contribute to this package.

The documentation covers the details of the automatic pagination support: https://slack.dev/node-slack-sdk/web_api#pagination. The first example could be used to get the complete list of channels if you simply replace web.conversations.history() with web.conversations.list().

We don't recommend using this technique anymore, because its very rare that you actually need the whole list. In fact, automatic pagination will be removed in the next major version (v5.0.0) which will be released soon. But if this is really what you want to do (as indicated in your question), then you should look to the second example in that section. I've copied it below:

const WebClient = require('@slack/client');

// An access token (from your Slack app or custom integration - xoxp, or xoxb)
const token = process.env.SLACK_TOKEN;

const web = new WebClient(token);

async function getAllChannels(options) 
 async function pageLoaded(accumulatedChannels, res) 
 // Merge the previous result with the results in the current page
 const mergedChannels = accumulatedChannels.concat(res.channels);

 // When a `next_cursor` exists, recursively call this function to get the next page.
 if (res.response_metadata && res.response_metadata.next_cursor && res.response_metadata.next_cursor !== '') 
 // Make a copy of options
 const pageOptions = ...options ;
 // Add the `cursor` argument
 pageOptions.cursor = res.response_metadata.next_cursor;

 return pageLoaded(mergedChannels, await web.conversations.list(pageOptions));
 

 // Otherwise, we're done and can return the result
 return mergedChannels;
 
 return pageLoaded([], await web.conversations.list(options));


(async () => 
 const allChannels = await getAllChannels( exclude_archived: true, types: 'public_channel' );
 console.log(allChannels);
)();

PS. In version v5.0.0, we're planning to include a helper method that will make this much simpler. For now, the second example is the most forward-compatible way, and I recommend that you refactor it to use the helper once v5.0.0 is released.