Our first approach for solving Melzak's Problem is derived by analogy from the proofs provided in [B01] to show that Right Regular Prisms minimize FF over all prisms. More specifically, given a polyhedron P we define a variationP_{t} of P to be a continuous deformation of P so that F(Pt)F(Pt) is a differentiable function of t with P0=P.P0=P. We then calculate
d
d
t
F
(
P
t
)

t
=
0
,
d
d
t
F
(
P
t
)

t
=
0
,
(2)in order to see whether F(Pt)F(Pt) has a critical point, and possibly a minimum, at t=0.t=0.
There are some difficulties with this approach. First, as a theoretical matter, although it is true that if P is the solution to Melzak's Problem then F(Pt)F(Pt) achieves a minimum at t=0t=0 for any variation P_{t} of P,P, we may encounter a pseudominimizer, a polyhedron P which has ddtF(Pt)t=0=0ddtF(Pt)t=0=0 for all variations P_{t} of P which is only locally a solution to Melzak's Problem. Second, as a practical matter, finding variations of a given polyhedron is typically delicate, as edges tend to appear and disappear as we deform polyhedra.
Given the idea of taking variations of polyhedra, the first task was to verify that ddtF(RTPt)t=0ddtF(RTPt)t=0 for any variation RTPtRTPt of the Regular Triangular Prism. We checked this for two variations, which are not merely deformations on the Regular Triangular Prism into other prisms. The first variation, consisted of lifting a vertex of the triangular top, to get a polyhedron Pθ,Pθ, where θ is the angle between the old triangular top and the new one:
A calculation shows that
F
(
P
θ
)
=
2
(
2
sec
θ
+
10
+
2
tan
θ
)
3
2
+
tan
θ
F
(
P
θ
)
=
2
(
2
sec
θ
+
10
+
2
tan
θ
)
3
2
+
tan
θ
(3)which attains its minimum at θ=0.θ=0.
Next, we varied the Regular Triangular Prism by taking the triangular base and expanding it, to form a tetrahedron with the top sliced off:
Taking the base to be equilateral triangle, then if 1+n1+n was the length of an edge of the base, then we have for the variation P_{n}
F
(
P
n
)
=
(
6
+
3
n
+
3
(
3
4
)
n
2
+
1
)
3
3
12
n
(
(
1
+
n
)
3
)

1
F
(
P
n
)
=
(
6
+
3
n
+
3
(
3
4
)
n
2
+
1
)
3
3
12
n
(
(
1
+
n
)
3
)

1
(4)which has minimum at n=0,n=0, when the figure is a Regular Triangular Prism.
These calculations are further evidence that the Regular Triangular Prism is the proposed minimizer. On the other hand, taking variations of the cube showed that it is a pseudominimizer. First, we vary the cube by lifting the top via two vertices:
When θ is the angle between the old top and the new one, then for the variation P_{θ} we have
F
(
P
θ
)
=
(
10
+
2
sec
θ
+
2
tan
θ
)
3
1
+
tan
θ
2
,
F
(
P
θ
)
=
(
10
+
2
sec
θ
+
2
tan
θ
)
3
1
+
tan
θ
2
,
(5)which has a minimum at θ=0.θ=0.
Next, we varied the cube by fixing a vertex, lifting the diagonal vertex a certain height and the two adjacent vertices by half that height:
In this case, when θ was the angle between the old top and the new one, then for the variation P_{θ} we have:
F
(
P
θ
)
=
(
8
+
2
2
tan
θ
+
4
2
1
cos
2
θ
+
1
)
)
3
1
+
2
2
tan
θ
,
F
(
P
θ
)
=
(
8
+
2
2
tan
θ
+
4
2
1
cos
2
θ
+
1
)
)
3
1
+
2
2
tan
θ
,
(6)which has minimum at θ=0.θ=0.
The next variation of the cube is collapsing two of the sides together, so as to form a shape approaching the Regular Triangular Prism:
When θ is the angle of collapse, then
F
(
P
θ
)
=
(
8
+
4
cos
θ

4
tan
θ
)
3
1

tan
θ
,
F
(
P
θ
)
=
(
8
+
4
cos
θ

4
tan
θ
)
3
1

tan
θ
,
(7)which has minimum at θ=0,θ=0, when the figure is a cube. Observe that although the figure with θ=π6θ=π6 is the Regular Triangular Prism, F(Pθ)π6F(Pθ)π6 with the formula given above is not F(RTP).F(RTP). This is because when θ=π6,θ=π6, two vertical edges have collapsed into one, and our formula above double counts these edges at θ=π6.θ=π6.