Is it possible to proxy a page and replace some content with content that is dynamically fetched from an API?

XDN Q&A
#1

See the comment in the code below. Is that possible?
// Inject 'content' where content comes from a dynamic call to an API

.match("/example-react-progressive-web-apps-ecommerce/", ({ proxy }) => {
    proxy("origin", {
      path: "/company",
      transformResponse: (res, req) => {
        const $ = cheerio.load(res.body)
        // Remove everything but the header and footer
        $(".section").remove()
        // Insert our new div that will house our new content
        $(".navbar-wrapper").after('<div id="main-content-container"></div>')
        // Change page title
        $("title").text("Example eCommerce PWAs")
        // Rewrite links in original page
        $("a").each(function () {
          let old_href = $(this).attr("href")
          if (old_href[0] === "/") {
            let new_href = "https://www.moovweb.com" + old_href
            $(this).attr("href", new_href)
          }
        })

        // Inject 'content' where content comes from a dynamic call to an API
        $('#main-content-container').append(content)

        res.body = $.html()
        res.setHeader("content-length", res.body.length)
      },
    })
  })
1 Like
#2

@mark.brocato @ianand @ierceg, I believe this is a feature you guys used to call MUR?

#3

@ajay Your use case is one of the intended use cases of transformResponse. Working from your example and making it as optimal as possible, you would first issue the dynamic call and while waiting for that I/O to come back you would process the HTML you already have (notice the added async in the function declaration):

.match("/example-react-progressive-web-apps-ecommerce/", ({ proxy }) => {
    proxy("origin", {
      path: "/company",
      transformResponse: async (res, req) => {
        // Issue the dynamic call first.
        const dynamicResponsePromise = fetch('https://maybe-another-upstream.com/dynamic-call')

        const $ = cheerio.load(res.body)
        // Remove everything but the header and footer
        $(".section").remove()
        // Insert our new div that will house our new content
        $(".navbar-wrapper").after('<div id="main-content-container"></div>')
        // Change page title
        $("title").text("Example eCommerce PWAs")
        // Rewrite links in original page
        $("a").each(function () {
          let old_href = $(this).attr("href")
          if (old_href[0] === "/") {
            let new_href = "https://www.moovweb.com" + old_href
            $(this).attr("href", new_href)
          }
        })

        // We have processed the HTML so now await the dynamic response.
        const dynamicResponse = await dynamicResponsePromise
        // TODO: add error checking and data validation of the dynamic response
        // Now await the content of the dynamic response.
        const content = await dynamicResponsePromise.text()

        // Inject 'content' where content comes from a dynamic call to an API
        $('#main-content-container').append(content)

        res.body = $.html()
        res.setHeader("content-length", res.body.length)
      },
    })
  })

We used to call it MUR as in “Multiple Upstream Requests”.

1 Like
#4

Note that actions in transformResponse will be performed at the serverless tier and not the edge.

Also, in situations like these you may find it helpful to leverage our serverless logging feature for debugging as described here.

2 Likes
#5

@ierceg I have tried your way.
It worked in local well, but it doesn’t work when deployed.
The error is issued when issuing the dynamic call first.

 const dynamicResponsePromise = fetch('https://maybe-another-upstream.com/dynamic-call')

at this part, the error is issued like the following:

ERROR: XDN error - Error in ResponseWriter: {}
#6

@jianxing Sorry to hear that. Is the error message returned in the response body?

#7

I have solved this issue by using axios instead of fetch.

#8

That is interesting. Do you mind posting the code using axios?

#9

I used axios to fetch the data from external API.
It works well not only in local, but also when deployed.

 .match('/', ({ proxy }) => {
    proxy('origin', { 
      path: '/',
      transformResponse: async (res, req) => {
        const $ = cheerio.load(res.body)

        //fetching the data from API
        const data = await axios.get("https://maybe-another-upstream.com")
.....
#10

And with the exact same code await fetch was throwing an error?

#11

yes, it throw an error when deployed

#12

Ok, thank you - we will investigate.

#13

Where did you import fetch from? Fetch is globally available in browsers, but in node you need to use something like node-fetch or cross-fetch.