I believe it's O(n2) , and it complies with observed result.
/*******Updated Analysis**********/
As innermost loop only executes on the conditions that i ≦ n0.5 and j ≦ n0.5, the original pseudocode can be divided to two parts.
Part 1:
for i = 1 to n0.5
for j = i to n0.5
for k = j*j to n
sum++;
Part 2:
for i = 1 to n
for j = n0.5 to n
for k = j*j to n //will only run once and end loop, hence O(1)
For the first part
The innermost loop (sum++) execute on every combination that satisfies 1 ≦ i ≦ j ≦ k0.5 ≦ n0.5
Hence the total count should be something like n0.5 * n0.5 * n giving the time complexity O(n2)
And the second part is clearly O(n2), hence the original algorithm also run at O(n2).