Hi Wenjun,
Let's take a look at the external loop first. For j, there is obviously n loops because it starts from 1 and increase by 1 every time until n.
Then internally, let's look inside at every loop of j. k = k + n^1/3, that is, k starts from j and increase by n^1/3 every time until n. We can suppose k = n when it stops (since it will only be 1 time more or less increment). How many times did k increase? Let it be x times. We have k + x*n^1/3 = n, so x = (n-k) / n^1/3 = n^2/3 - k / n^1/3.
This polynomial equation is the number of loops of k, for every j. The first element does not change for each j loop while the second does (as k starts from j for every j loop). Since we have n loops of j, so the total number of loops we have is actually
n*(n^2/3) - (1+2+3+...+n-1) / n^1/3
= n^5/3 - (n*(n-1)/2) / n^1/3
= n^5/3 - (1/2)*n^5/3 + (1/2)*n^2/3
Since every loop takes constant time here, we have the time complexity as:
= O((1/2)*n^5/3) = O(n^5/3)
BTW, I think using O() earlier is more convenient when we do it for ourselves.