Continuing from your solution,

b = 5-8a   ——– (1)

h(3) = 3a+b

h^-1 (3) = (3-b)/a

Given h(3) = h^-1 (3)

3a+b = (3-b)/a    => 3a² +ab +b-3 = 0     ——-(2)

Substitute (1) into (2),

3a²  + a(5-8a) + (5-8a) -3 = 0

3a²  + 5a – 8a² + 5 – 8a – 3 = 0

-5a² -3a + 2= 0

5a² + 3a – 2 = 0

(5a – 2) (a + 1) = 0

=> a = 2/5  or  a = -1  but it is given that a <> -1 

Hence a = 2/5 =0.4 ,   b = 5-8(2/5) =  (9/5) or write it in mixed number 1  4/5  or  1.8

Please click “Accept Answer” if the answer is correct.